Disable istio sidecar injection in Suggestion and Training Jobs #1050
Conversation
|
Will this solve the problem without adding to the training pods? Have you tried? Btwn, unstructured is just a map |
No, we must add it to training jobs. The main problem happens when we download datasets in training job behind istio-proxy.
Ok, in that case, we can work only with 3 supported job type (v1Job, TFJob and PytorchJob) and I will try to add annotation to them before submitting. |
|
I added few changes to this PR. Instead of changing each example yaml file, I add istio sidecar annotation in the Katib controller. In that case, I add annotation to the right place depends on JobKind and user can't submit any other job than If we don't want to limit user with these types of Training Jobs, we can run script with annotation only for these three types. What do you think @richardsliu @gaocegege @hougangliu @johnugeorge ? |
|
/retest |
4 similar comments
|
/retest |
|
/retest |
|
/retest |
|
/retest |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: johnugeorge The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Fixes: kubeflow/kubeflow#4742.
I added part in Katib controller to create Suggestion deployment with
sidecar.istio.io/inject: falseannotation.I am not sure that we can add this annotation to training jobs in the controller since we submit job from
Unstructuredobject: https://github.com/kubeflow/katib/blob/master/pkg/controller.v1alpha3/trial/trial_controller.go#L271.Because of that, I added annotations to template in each example.
/assign @johnugeorge @richardsliu
/cc @hougangliu @gaocegege @krishnadurai
This change is